Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition1

Functional magnetic resonance imaging (fMRI) studies of emotion, personality, and social cognition have drawn much attention in recent years, with high-profile studies frequently reporting extremely high (e.g., >.8) correlations between brain activation and personality measures. We show that these correlations are higher than should be expected given the (evidently limited) reliability of both fMRI and personality measures. The high correlations are all the more puzzling because method sections rarely contain much detail about how the correlations were obtained. We surveyed authors of 55 articles that reported findings of this kind to determine a few details on how these correlations were computed. More than half acknowledged using a strategy that computes separate correlations for individual voxels and reports means of only those voxels exceeding chosen thresholds. We show how this nonindependent analysis inflates correlations while yielding reassuring-looking scattergrams. This analysis technique was used to obtain the vast majority of the implausibly high correlations in our survey sample. In addition, we argue that, in some cases, other analysis problems likely created entirely spurious correlations. We outline how the data from these studies could be reanalyzed with unbiased methods to provide accurate estimates of the correlations in question and urge authors to perform such reanalyses. The underlying problems described here appear to be common in fMRI research of many kinds—not just in studies of emotion, personality, and social cognition. Functional magnetic resonance imaging (fMRI) studies of emotion, personality, and social cognition scarcely existed 10 years ago, and yet the field has already achieved a remarkable level of attention and prominence. Within the space of a few years, it has spawned several new journals (Social Neuroscience, Social Cognitive and Affective Neuroscience) and is the focus of substantial new funding initiatives (National Institute of Mental Health, 2007) while receiving lavish attention from the popular press (Hurley, 2008) and the trade press of the psychological research community (e.g., Fiske, 2003). Perhaps even more impressive, however, is the number of papers from this area that have appeared in such prominent journals as Science, Nature, and Nature Neuroscience. Although the questions and methods used in such research are quite diverse, a substantial number of widely cited papers in this field have reported a specific type of empirical finding that appears to bridge the divide between mind and brain: extremely high correlations between measures of individual differences relating to personality, emotion, and social cognition and measures of brain activity obtained using fMRI. We focus on these studies here because this was the area where these correlations came to our attention; we have no basis for concluding that the problems discussed here are necessarily any worse in this area than in some other areas. The following are a few examples of many studies that will be discussed in this article: This article was formerly known as ‘‘Voodoo Correlations in Social Neuroscience.’’ Address correspondence to Harold Pashler, Department of Psychology 0109, University of California, San Diego, La Jolla, CA 92093; e-mail: [email protected]. Studies of the neural substrates of emotion, personality, and social cognition rely on many methods besides fMRI and positron emission tomography, including electroencephalography and magnetoencephalography, animal research (e.g., cross-species comparisons), and neuroendocrine and neuroimmunological investigations (Harmon-Jones & Winkielman, 2007). PERSPECTIVES ON PSYCHOLOGICAL SCIENCE 274 Volume 4—Number 3 Copyright r 2009 Association for Psychological Science 1. Eisenberger, Lieberman, and Williams (2003), writing in Science, described a game they created to expose individuals to social rejection in the laboratory. The authors measured the brain activity in 13 individuals while the actual rejection took place, and they later obtained a self-report measure of how much distress the subject had experienced. Distress was correlated at r 5 .88 with activity in the anterior cingulate cortex (ACC). 2. In another Science article, Singer et al. (2004) found that the magnitude of differential activation within the ACC and left insula induced by an empathy-related manipulation displayed a correlation between .52 and .72 with two scales of emotional empathy (the Empathic Concern Scale of Davis and the Balanced Emotional Empathy Scale of Mehrabian). 3. Writing in NeuroImage, Sander et al. (2005) reported that a subject’s proneness to anxiety reactions (as measured by an index of the Behavioral Inhibition System), correlated at r 5 .96 with the difference in activation of the right cuneus between attended and ignored angry speech. In this article, we will discuss many studies reporting similar sorts of correlations. The work that led to the present article began when we became puzzled about how such impressively high correlations could arise. We describe our efforts to resolve this puzzlement, and the conclusions that our inquiries have led us to. Why should it be puzzling to find high correlations between brain activity and social and emotional measures? After all, if new techniques are providing a deeper window on the link between brain and behavior, does it not make sense that researchers should be able to find the neural substrates of individual traits and thus potentially reveal stronger relationships than have often been found in purely behavioral studies? The problem is this: It is a statistical fact (first noted by researchers in the field of classical psychometric test theory) that the strength of the correlation observed between Measures A and B (rObservedA,ObservedB) reflects not only the strength of the relationship between the traits underlying A and B (rA,B), but also the reliability of the measures of A and B (reliabilityA and reliabilityB, respectively). In general, rObservedA; ObservedB 1⁄4 rA;B ðreliabilityA reliabilityBÞ Thus, the reliabilities of two measures provide an upper bound on the possible correlation that can be observed between the two measures (Nunnally, 1970). RELIABILITY ESTIMATES So what are the reliabilities of fMRI and personality and emotional measures likely to be? The reliability of personality and emotional scales varies between measures and according to the number of items used in a particular assessment. However, test– retest reliabilities as high as .8 seem to be relatively uncommon and are usually found only with large and highly refined scales. Viswesvaran and Ones (2000) surveyed many studies on the reliability of the Big Five factors of personality, and they concluded that the different scales have reliabilities ranging from .73 to .78. Hobbs and Fowler (1974) carefully assessed the reliability of the subscales of the MMPI and found numbers ranging between .66 and .94, with an average of .84. In general, a range of .7 to .8 would seem to be a somewhat optimistic estimate for the smaller and more ad hoc scales used in much of the research described below, which could well have substantially lower reliabilities. Less is known about the reliability of blood oxygenation level dependent (BOLD) signal measures in fMRI, but some relevant studies have recently been performed. Kong et al. (2006) had subjects engage in six sessions of a finger-tapping task while recording brain activation. They found test–retest correlations of the change in BOLD signal ranging between 0 and .76 for the set of areas that showed significant activity in all sessions. Manoach et al. (2001, their Fig. 1, p. 956) scanned subjects in two sessions of performance with the Sternberg memory scanning task and found reliabilities ranging between .23 to .93, with an average of .60. Aron, Gluck, and Poldrack (2006) had people perform a classification learning task on two separate occasions widely separated in time and found voxel-level reliabilities with modal values (see their Fig. 5, p. 1005) a little below .8. Johnstone et al. (2005, p. 1118) examined the stability of amygdala BOLD response to presentations of fearful faces in multiple sessions: Intraclass correlations for the left and right amygdale regions of interest were in the range of .4 to .7 for the two sessions (which were separated by 2 weeks). Thus, from the literature that does exist, it would seem reasonable to suppose that fMRI measures computed at the voxel level will not often have reliabilities greater than about .7. This is the case because the correlation coefficient is defined as the ratio between the covariance of two measures and the product of their standard deviations: rx;y 1⁄4 sxy sxsy. Real-world measurements will be corrupted by (independent) noise, thus the standard deviations of the measured distributions will be increased by the additional noise (with a magnitude assessed by the measure’s reliability). This will make the measured correlation lower than the true underlying correlation by a factor equal to the geometric mean of reliabilities. We consider test–retest reliabilities here (rather than interitem, or split–half reliability) because, for the most part, the studies we discuss gathered behavioral measure at different points in time than the fMRI data. In any case, internal reliability measures, like coefficient alpha, do not generally appear to be much higher in this domain. We focus here on studies that look at the reliability of BOLD signal activation measures rather than the reliability of patterns of voxels exceeding specific thresholds, which tend to be substantially lower (e.g., Stark et al., 2004). It seems likely that restricting the reliability analysis to regions consistently active in all sessions would tend to overestimate the reliability of BOLD signal in general. They found somewhat higher reliabilities for voxels within a frontostriatal system that they believed was most specifically involved in carrying out the probabilistic classification learning. Volume 4—Number 3 275 Edward Vul, Christine Harris, Piotr Winkielman, and Harold Pashler